When we first consolidated dotnet/coreclr, dotnet/corefx, dotnet/core-setup, and mono/mono in .NET 5, we brought in separate product build, test build, and test run pipeline jobs, as before consolidation we were seeing increased build throughput and reliability by having separate jobs, ensuring we built each asset exactly once, and a complex matrix of how they interacted.

Over time, we've removed a few of these jobs and allowed double-building to reduce dependencies:

• Libraries removed their separate "build test" job.

• The CoreCLR/Mono "build test" job now builds the assets it needs directly instead of depending on other jobs.

• The Libraries job now builds System.Private.CoreLib in-job (and now can even just build the ref-assembly for CoreLib)
   
  Additionally, we've moved many jobs to build the product (and sometimes test) in one job using the "Global build" template:

• All Mono devices jobs use the "global build" to build the product (and build runtime tests if needed)

• All NativeAOT jobs use the "global build" template

• Nearly all community-supported legs build using the "global build" template.

• A few outerloop pipelines use the "global build" template for simplicity.

Today, our pipelines are split between using the "Global build" template that directly invokes the root build script to build the product in one command and the separate "build job" templates for building specific subsets (CoreCLR, Libraries, Mono, Installer) separately.

This has added significant complexity in many of our YAML scripts to support both cases and confusion about when to use one scenario or the other. For example, some jobs in the "runtime" pipeline use the global build job template to build the product and run tests whereas others don't.

Recently, we moved the official build to use the "global build" template for every job as a) we needed to move in this direction for the VMR and b) we believed that due to job queue time and upload/download time, we'd save on build execution time.

We saw decent improvements, as well as a significant reliability boost by merging these jobs together in the official build.

This PR takes us the next step: Merge all CoreCLR, Mono, and Libraries product build steps (and installer build steps when we aren't running tests) into single jobs instead of being split into 2 or 3 jobs (as all of the Mono runs also required some components of the corresponding CoreCLR build).

In addition, to avoid ending up in a split situation where many of our outerloop legs are still on the older system while PR and official builds are all on the new system, I have moved every pipeline to the "global build" model, not just the PR pipeline.

Job Dependency Graph Changes

CoreCLR + Libraries + Installer (w/o Installer Tests)

flowchart TD

subgraph Old Model
direction LR
CLR[CoreCLR Build Job]
Libs[Libraries Build Job]
Host[Installer Build Job]
CLR --> Host
Libs --> Host
BRT[Build Runtime Tests] --> RT
CLR --> RT[Runtime Tests]
Libs --> RT
CLR --> LT[Libraries Tests]
Libs --> LT
end
subgraph New Model
Global[Global build job: CLR + Libs + Host + Packs]
GBRT[Build Runtime Tests]
GRT[Runtime Tests]
GLT[Libraries Tests]
Global --> GRT
Global --> GLT
GBRT --> GRT
end

CoreCLR + Libraries + Installer w/ Tests

flowchart TD

subgraph Old Model
direction LR
CLR[CoreCLR Build Job]
Libs[Libraries Build Job]
Host[Installer Build and Test]
CLR --> Host
Libs --> Host
BRT[Build Runtime Tests] --> RT
CLR --> RT[Runtime Tests]
Libs --> RT
CLR --> LT[Libraries Tests]
Libs --> LT
end
subgraph New Model
Global[Global build job: CLR + Libs]
GI[Installer Build and Test]
GBRT[Build Runtime Tests]
GRT[Runtime Tests]
GLT[Libraries Tests]
Global --> GRT
Global --> GLT
GBRT --> GRT
Global --> GI
end

Mono Runtime Tests

flowchart TD

subgraph Old Model
direction LR
CLR[CoreCLR Build Job, all of CoreCLR]
Libs[Libraries Build Job]
Mono[Mono Build Job]
BRT[Build Runtime Tests] --> RT
CLR --> RT[Runtime Tests]
Libs --> RT
Mono --> RT
end
subgraph New Model
Global[Global build job: Mono + CoreRun + ilasm/dasm + Libs]
GBRT[Build Runtime Tests]
GRT[Runtime Tests]
Global --> GRT
GBRT --> GRT
end

YAML Template Flow Changes

In addition to the job simplifications above, there's also some YAML template simplification as well. In addition to removing the build-coreclr-and-libraries-job template, I consolidated the general flow of the JIT outerloop pipelines for CoreCLR tests into a single pipeline template, simplifying defining them.

Additionally, I moved all outerloop pipelines that run libraries tests to use the global build job with the "Send to Helix" extra step instead of shuffling files between jobs.

Q&A

• Why do we still have separate jobs per vertical in the PR pipeline?
  • We don't want to wait to run the runtime tests until the libraries tests pass or vice versa. Additionally, building the runtime tests still takes a while (~30 minutes), so we don't want to make that a requirement before any tests start to run until we've reduced that time.
• Why are installer tests still in a separate job?
  • Installer tests are very quick, but we don't want to block libraries and runtime tests from running if installer tests fail?
• Why is installer build still in the installer test job?
  • It's easier given current infrastructure to keep the "Installer Build and Test" job as-is rather than moving the building steps out entirely.

For validation, I'm happy to run any requested pipelines on this PR. I obviously can't run them all, but I'm happy to run a good spread to validate that I didn't break anything (I've been running various pipelines throughout working on this to validate the experience as I went).